Mass storage system and method using hard disk and solid-state media

ABSTRACT

Methods and systems for mass storage of data over two or more tiers of mass storage media that include nonvolatile solid-state memory devices, hard disk devices, and optionally volatile memory devices or nonvolatile MRAM in an SDRAM configuration. The mass storage media interface with a host through one or more PCIe lanes on a single printed circuit board.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation-in-part patent application of co-pending U.S.patent application Ser. No. 12/713,349, filed Feb. 26, 2010, whichclaimed the benefit of U.S. Provisional Application No. 61/162,488,filed Mar. 23, 2009. This application is also related to co-pending U.S.patent application Ser. No. 12/815,661, filed Jun. 15, 2010, whichclaimed the benefit of U.S. Provisional Application No. 61/218,571,filed Jun. 19, 2009. The contents of these prior applications areincorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention generally relates to mass storage devices for usewith host systems, including computers and other processing apparatuses.More particularly, this invention relates to a PCIe-based mass storagesystem that utilizes a hybrid drive comprising at least one highcapacity hard disk component for low-frequency accessed data in a hostsystem, along with at least one nonvolatile solid-state component forsystem data and intermediate storage of high-frequency accessed data inthe host system.

Nonvolatile memory subsystems and mass storage devices of moderncomputers are typically addressed through the system bus using thesouthbridge or any equivalent logic, for example, an input/output (I/O)controller hub (ICH) introduced by Intel Corporation. A deviation fromthis scheme is the PCI express (PCIe) bus having branches originatingfrom either node of the system core logic, including integrated systemagents or un-cores embedded into modern central processing unit (CPU)dies.

PCIe has become the fast system interconnect bus of choice, and offersin its latest generation (V3.x) 1 GB/s bandwidth in each direction.Particularly the branches tied to the first node in the interconnectcascade further have ultra low latency which makes them extremelysuitable for any kind of data access. Another advantage of the PCIeinterconnect or any expansion slot implementation is that it offers theuser a high degree of freedom with respect to populating the system withperipheral devices including nonvolatile memory devices or mass storagemedia.

In the past, storage-related add-on cards were typically small computersystem interface (SCSI) or RAID controllers with better performance orricher feature sets than on-board host bus adapters for parallel orserial ATA devices attached via cables to the card. However, as aconsequence of the miniaturization of drive technology, including weightreduction, and the introduction of solid-state media, separation andcable connectivity between the interface and the actual storage mediaare no longer a prerequisite. A consequent trend has been thedevelopment of new form factors, including the integration of the datacarrier onto an interface card along with the control logic for astreamlined, cable-less and ultra-compact device. For a number ofreasons, specifically relating to weight, space constraints and power,this type of integrated PCIe-based storage device has mostly beenrestricted to the use of solid-state media.

Solid-state media, particularly NAND flash memory devices, have theadvantage of allowing random access of data in the array over severalparallel channels, and therefore it is far superior to rotatable mediawith respect to access speed, I/O switching and, moreover, sustaineddata transfer rates. On the downside, however, NAND flash memory cannotcompete with rotatable media on cost per bit, nor with respect to dataretention. Specifically, whereas magnetic media such as rotatableplatters have practically unlimited data retention, NAND flash cellslose data through stress-induced leakage current during normaloperation, as well as simple diffusion of electrons from the floatinggate through the gate oxide layer into the substrate during normaloperation and when the device is powered down.

In contrast to initial expectations regarding solid-state drives (SSD)becoming a complete replacement of hard disk drives (HDD) as the massstorage media used in computers, what has emerged in practice is afunctional dichotomy between solid-state drives and hard disk drives.SSDs are gaining acceptance in any function or role warranting frequentaccess of data. This particular scenario not only fits the strength ofSSDs in delivering high I/O throughput, but further adds the benefitthat, because of frequent accesses, data integrity can be easilymonitored through the bit error rate of a block or page on any givenread access. Any increase in the bit error rate can then be used as anindicator for creeping data deterioration through leakage current.Understandably, monitoring bit error rates requires read access of thedata, which puts rarely accessed files at a greater risk of corruptionthan files that are frequently monitored. In so far, NAND flash memorymay not be the optimal storage medium.

The other end of the spectrum focuses on archiving of rarely accesseddata. The majority of this type of data tends to be multimedia filessuch as photographs or movie clips, or archived documents includingpersonal data, records or even e-books that are collected after purgingthem from the reader of choice. This type of data may not be accessedfor months or years and, while it would be possible to move them to anoffline vault or burn them to optical media, it is more in line with thedigital life style to have the archive available at any time, forexample in the form of a centralized server.

For a simplified design of any such centralized media and documentserver, it would be highly advantageous to have a solution that reducesspace requirements by providing a highly integrated device featuringhybrid storage technology in combination with intelligent datamanagement to combine the best features of both solid-state androtatable (hard disk) media with respect to access speed, I/Operformance and data retention at the lowest cost per bit.

BRIEF DESCRIPTION OF THE INVENTION

The present invention provides methods and systems capable ofcapitalizing on fast access capabilities (low initial access latencies)of nonvolatile solid-state memory technologies currently available foruse in host systems (including computers and other processingapparatuses), in combination with large capacity electromechanicalstorage devices with minimal degradation of data over time.

According to a first aspect of the invention, an integrated PCIe-basedmass storage system is provided comprising a printed circuit board thatis electrically compatible with a PCIe interface, an edge connector onthe printed circuit board, at least one solid-state mass storage devicehaving nonvolatile solid-state memory components and at least a firstcontroller that interfaces with the nonvolatile solid-state memorycomponents, at least one hard disk mass storage device having at leastone hard disk drive with a rotatable platter, and a RAID controller thatinterfaces with the hard disk mass storage device. The mass storagesystem is configured so that the solid-state mass storage device storesfiles with a read access frequency above a threshold, the hard disk massstorage device stores files with a read access frequency below thethreshold, and if the read access frequency of a file stored on the harddisk mass storage device increases above the threshold, a copy of thefile is written to the solid-state mass storage device.

According to a second aspect of the invention, a method is provided forpermanently storing data on an integrated PCIe-based mass storage systemcomprising a printed circuit board that is electrically compatible witha PCIe interface, an edge connector on the printed circuit board, atleast one solid-state mass storage device having nonvolatile solid-statememory components and at least a first controller that interfaces withthe nonvolatile solid-state memory components, at least one hard diskmass storage device having at least one hard disk drive with a rotatableplatter, a RAID controller that interfaces with the hard disk massstorage device, and optionally at least one memory module having memorycomponents associated with the solid-state mass storage device. Themethod includes determining a frequency of access for a file stored onthe hard disk mass storage device and copying the file to thesolid-state mass storage device if the access frequency is above athreshold, and then updating the access path or metadata for the file topoint to the solid-state mass storage device.

According to a third aspect of the invention, an integrated PCIe-basedmass storage system is provided comprising a printed circuit board thatis electrically compatible with a PCIe interface, an edge connector onthe printed circuit board, at least one volatile memory module havingvolatile memory components, at least one solid-state mass storage devicehaving nonvolatile solid-state memory components and at least a firstcontroller that interfaces with the nonvolatile solid-state memorycomponents, at least one hard disk mass storage device having at leastone hard disk drive with a rotatable platter, and a RAID controller thatinterfaces with the hard disk mass storage device. The mass storagesystem is configured so that the volatile memory module is a cache forthe solid-state mass storage device, the solid-state mass storage devicepermanently stores data with an access frequency above a threshold, thehard disk mass storage device stores data with an access frequency belowthe threshold.

According to another aspect of the invention, a method is provided forpermanently storing data on an integrated PCIe-based mass storage systemcomprising a printed circuit board, an edge connector adapted tointerface with a PCIe expansion slot, first and second RAID controllersinterfacing with the edge connector through a PCIe switch, at least twosolid-state mass storage devices in a striped array and each solid-statemass storage device having nonvolatile solid-state memory componentsfunctionally connected to the first RAID controller, multiple hard diskmass storage devices functionally connected to the second RAIDcontroller and each hard disk mass storage devices having at least onehard disk drive with a rotatable platter, and a 64-bit wide DRAM memorymodule as cache for the solid-state mass storage devices. The memorymodule is configured into a functional upper 32-bit block and a lower32-bit block, with each of the upper and lower 32-bit blocks havingseparate command and address buses. The upper 32-bit block is a cachefor a first of the solid-state mass storage devices and the lower 32-bitblock is a cache for a second of the solid-state mass storage devices.The method includes copying a file accessed from the hard disk massstorage devices to one of the caches provided by the memory module andupdating the access path for the file to point to the solid-state massstorage devices, and storing a copy of the file in the nonvolatilesolid-state memory components of the solid-state mass storage devices ifthe host system writes the file back to the mass storage system;determining a frequency of access of the file; storing the file to thehard disk mass storage devices if the frequency of access drops below athreshold; and deleting the file from the solid-state mass storagedevice.

A technical effect of this invention is the ability of the mass storagesystem to combine advantageous aspects of different types of nonvolatilemass storage media to create multiple tiers of storage. Commitment ofindividual blocks of data into such a multi-tiered mass storage systemcan be managed by a hierarchical storage management (HSM)implementations of types known in the art.

Other aspects and advantages of this invention will be betterappreciated from the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically represents a PCIe-based mass storage system in amodular configuration that includes a carrier board, a RAID controller,and solid-state drives and hard disk drives that are removably mountedto the carrier board to define a two-tiered storage system.

FIG. 2 schematically represents a simplified perspective view of thestorage system of FIG. 1 with three drives mounted at its front and backfaces of its carrier board.

FIG. 3 schematically represents an embodiment of a PCIe-based massstorage system, in which two RAID controllers are implemented on thesame carrier board of the mass storage system, and each RAID controlleroccupies its own set of PCIe lanes.

FIG. 4 schematically represents another embodiment of a PCIe-based massstorage system, in which an array of hard disk drives, a solid-statedrive comprising an array of nonvolatile memory components, and an arrayof volatile memory components are directly mounted on a carrier board ofthe mass storage system to define a three-tiered storage system

FIG. 5 shows a similar embodiment as FIG. 4, but further including aPCIe switch adapted for interfacing with the system logic.

FIG. 6 schematically represents a similar embodiment of the invention asFIG. 4, but with a single controller in an eight-channel PCIeconfiguration with one array in striping configuration over two SSDs, asecond array constituting six HDDs in a striped configuration with orwithout parity, and a hierarchical storage management (HSM) manager.

DETAILED DESCRIPTION OF THE INVENTION

The current invention is directed to a system that combines advantageousaspects of different types of nonvolatile mass storage media, andparticularly nonvolatile solid-state mass storage devices andelectromechanical mass storage devices, within a fully integrated devicethat is compatible with the PCIe standard interface in personalcomputers and servers. In preferred embodiments, such a PCIe-based massstorage system uses nonvolatile memory technologies as a storage tier,and electromechanical mass storage devices such as rotatableplatter-based hard disk drives as another storage tier havingpractically unlimited data retention. Particularly preferred embodimentsof the invention further incorporate volatile memory technologies, forexample, DRAM (dynamic random access memory), that are characterized byultra-fast access times and extremely low error rates, to provide yetanother tier of storage. Commitment of individual blocks of data intosuch a three-tiered mass storage system can be managed by a hierarchicalstorage management (HSM) implementation of a type that is known in theart and can either run on system software or be implemented in hardware,for example, a field programmable gate array (FPGA) or applicationspecific integrated circuit (ASIC).

Current mass storage devices typically center on one distincttechnology. For example, hard disk drives (HDDs) use nonvolatile(permanent) magnetic platters, solid-state drives (SSDs) use nonvolatile(permanent) solid-state memory devices, and system memory use volatilememory devices that often serve as a disk cache. None of these differentmedia is perfect for all applications, and there are shortcomingsassociated with each. For example, volatile memory devices such as DRAMhave the fastest access times and lowest error rates of existing memorydevices, the volatile nature of the media results in data retention onlywhile power is supplied. Furthermore, implementation of a large(TeraByte) mass storage system based on DRAM would be cost prohibitiveand exceed the power budget of most host systems, including computersand other processing apparatuses. Nonvolatile memory devices such asNAND flash offer acceptable performance at a reasonable price point, yethave limitations relating to long-term data retention. Rotatableplatters of hard disk drives, which are accessed through mechanicalmovements of a read/write head over the tracks and sectors, havepractically unlimited data retention and are extremely cost efficient.However, rotatable platters are limited by slow access times and, sincethe data stream is strictly serial, there are limitations with respectto the overall data throughput.

The present invention provides a PCIe-based mass storage system thatuses at least two of the above-mentioned storage media, such that themass storage system can be referred to as a hybrid storage system. Inthe mobile sector, HDD/SSD hybrids already exist, for example, theSeagate Momentus® XT. However, such devices use a relatively smallamount of NAND flash memory in a manner similar to a prefetch/readcache. That is, writes are initially committed to the HDD platters and,on a read access, data are fetched into the NAND flash array.Consequently, the device does not offer much in terms of write I/Operformance increase.

FIGS. 1 and 2 schematically represent an embodiment of a PCIe-based massstorage system 10 having a modular configuration. The mass storagesystem 10 includes a carrier board 12, a RAID controller 14, a datacache 16 for data buffering, power circuitry 18 (including, for example,voltage regulator and capacitors), and a hybrid mass storage systemcomprising nonvolatile mass storage devices 20 that are removablymounted on the carrier board 12 of the storage system 10 with dockingconnectors 22 and retention clips 24. The carrier board 12 can be aprinted circuit board of the type commonly used as expansion cards andhost bus adapters (HBA). The data cache 16 is made up of volatile memorycomponents, preferably in the form of DRAM chips, and is used to bufferwrites and/or prefetch reads from the storage devices 20. The storagedevices 20 can have male connectors (not shown) by which they aredirectly plugged into the female docking connectors 22, which preferablyprovide both power and data connections for the storage devices 20.Alternatively, the storage devices 20 could have female connectors thatconnect to a male edge connector on the carrier board 12, for example,as disclosed in U.S. patent application Ser. No. 12/783,978.Particularly preferred docking connectors 22 are SATA connectors. Thesystem 10 is further represented as comprising a system expansion slotinterface in the form of a PCIe edge connector 26, which provides aninterface for an expansion slot of a motherboard (not shown) of a hostsystem. All data and power buses (not shown) are integrated into thecarrier board 12, and the edge connector 26 enables power to be suppliedto the mass storage system 10 and data exchange between the system 10and its motherboard. An auxiliary power connector 28 is also shown bywhich additional power can be supplied to the system 10, if so desired.Electrical connections (not shown) on the carrier board 12 can beachieved in accordance with conventional industry practices.

According to a preferred aspect of the invention, individual massstorage devices 20 of the mass storage system 10 comprise either anarray of solid-state drives that access nonvolatile memory componentsthrough a multi-channel interface, or one or more hard disk drives foruse as a large (for example, TeraByte) capacity storage repository. Thestorage capacity of the repository can be achieved with a single harddisk drive or with an array of individual HDD drives (units), preferablyin a RAID Level 5 configuration that, in combination with the RAIDcontroller 14, provides redundancy and allows for rebuilding of thearray without data loss in the case of failure of one of the drives. Theoverall design of the PCIe-based mass storage system 10 is representedas being modular, that is, each individual device 20 can be removablyinstalled on the system 10 in order to provide a flexible configurationthat facilitates maintenance of the system 10, including replacement ofdefective drives and upgrades to larger capacity drives. Alternatively,some or all of the devices 20 or their memory components could be fullyintegrated onto the carrier board 12 of the system 10.

The mass storage system 10 of FIGS. 1 and 2 can be referred to as atwo-tiered storage system, in which a first storage tier (level) of thesystem 10 is provided by at least one storage device 20 that usessolid-state technology (hereinafter, SSD storage device(s) 20), and asecond storage tier (level) of the system 10 is provided by at least onestorage device 20 that uses hard disk drive technology (hereinafter, HDDstorage device(s) 20). Arrays of NAND flash memory devices (not shown)are particularly preferred for use as the solid-state memory in the SSDstorage devices 20, though the use of other solid-state memorytechnologies is also foreseeable, for example, phase change memory,magnetic RAM, and magneto-resistive RAM. A storage device 20 made up ofNAND flash memory devices will typically feature a SATA or SAS interface(not shown) to communicate with the edge connector 26 of the carrierboard 12. The NAND flash memory devices can be accessed by amulti-channel NAND flash interface, which can be optionally capable ofcompression/decompression of data in hardware while implementingde-duplication algorithms. In the two-tiered configuration of FIGS. 1and 2, all data transfers received from the host system can be initiallywritten to the SSD storage device(s) 20. If a given set of files storedon a SSD storage device 20 is not accessed within a certain period oftime, the files are copied to an HDD storage device 20 of the system 10,after which the files can be purged from the NAND flash memory devicesof the SSD storage device 20, for example, using HSM algorithms that canbe implemented in system software or with dedicated hardware, forexample, an FPGA or ASIC.

In the simplest case, the time stamp of any file written to the SSDstorage device 20 can be logged to a dedicated file to keep track ofvital parameters, for example, when the file was written to the SSDstorage device 20, when it was first and last read from the SSD storagedevice 20, and the number of accesses during a defined time interval.Using the above-noted HSM functionality, the mass storage system 10 isfurther capable of executing HSM algorithms based on data accessfrequency analysis using the SSD storage devices 20 as a first storagetier and the HDD storage devices 20 as a second storage tier. Inparticular, the mass storage system 10 is configured so that files thathave read access frequencies above a predetermined threshold are storedon the SSD storage devices 20, and files that have read accessfrequencies below the predetermined threshold are stored on the HDDstorage device 20. If the read access frequency of a file stored on aHDD storage device 20 increases above the threshold, a copy of the fileis written to a SSD storage device 20. On the other hand, if the readaccess frequency of a file stored on an SSD storage device 20 decreasesbelow the threshold, a copy of the file is written to a HDD storagedevice 20 and the file is purged from the SSD storage device 20. Purgingin this respect can mean that the file is invalidated through ahierarchical storage manager or on the level of the file system. As soonas the file-related data on the SSD storage device 20 are invalidated,they can be subjected to garbage collection and a subsequent eraseprocess.

In some cases, it may be advantageous to write the access frequency of agiven file to a translation look-aside buffer, such as a contentaddressable memory (CAM). The access frequency then becomes the addressor access parameter to generate as an output the physical or logicaladdresses of files within the array of NAND flash memory devices of theSSD storage device 20 at or below the frequency entered as criterion. Asimple routine could set a threshold that automatically generates anaccess of the file tied to a purge command to write out the data to anHDD storage device 20 and mark the respective pages of the NAND eraseblock as stale. Through garbage collection and TRIM, any valid datawithin the several erase blocks containing stale data can be coalescedand the free blocks can be committed to erasing.

As represented in FIGS. 1 and 2, the mass storage system 10 has a singlelogical system interface and a single RAID controller 14 for all of thestorage devices 20 of the system 10. In the embodiment of FIG. 3, asimilar mass storage system 10 is represented as comprising twodifferent RAID controllers 14 on the carrier board 12. (For convenience,consistent reference numbers are used in FIG. 3 to identify elementsthat are the same or functionally equivalent to elements of FIGS. 1 and2). Another alternative is for the mass storage system 10 to use oneRAID controller 14 and a more conventional PCIe-based host bus adapter(not shown) in place of the second RAID controller 14 on the carrierboard 12. Each controller and/or host bus adapter 14 preferably occupiesits own set of PCIe lanes, such that mass storage system 10 appears toits host system as two distinct mass storage systems. For example, giventhe relatively low bandwidth required by hard disk drives, it maysuffice to use a single-lane PCIe interface for communication betweenthe host system and the controller 14 for the HDD storage devices 20,while four PCIe lanes are used for communication between the host systemand the controller (NAND flash controller) 14 of the SSD storage devices20. Other combinations of PCIe lanes are also possible. For example, itis conceivable that the controller 14 for the NAND flash portion (SSDstorage devices 20) of the system 10 may occupy eight PCIe lanes,whereas the controller 14 for the HDD portion (HDD storage devices 20)of the system 10 uses four PCIe lanes. In either case, if two separatecontrollers 14 are used, data written from the HDD to the SSD portions(or vice versa) go through the system interface (edge connector 26) inorder to be written to system memory before being written back to theintended storage device 20.

PCIe lane splitting as discussed above is supported by the existing PCIespecifications, though as an optional feature. Many implementations ofPCIe may lack support of this feature. For the purpose of bettercompatibility, a PCIe switch or arbitrator may be used to interface withall PCIe lanes of the system 10 at the host system interface level, andthen arbitrate the lanes either through physical splitting or throughtime-division multiplexing. Given the latencies of either HDDs or NANDflash-based SSDs, additional latencies associated with a PCIe switch orarbitrator would be considered negligible.

On the other hand, a single controller 14 can be used to access the HDDand SSD storage devices 20 as long as enough device connections areavailable, as would be the case, for example, in an 8-channel RAIDcontroller 14. For example, in this type of configuration, six HDDstorage devices 20 could be used in a RAID Level 5 configuration withdistributed parity to maximize capacity, while maintaining fullredundancy. At the same time, two channels could be dedicated to twoSATA 6.0 interfaces in a RAID 0 configuration, allowing for maximumsequential transfer rates of 1.2 GB to or from the SSD storage devices20. Another possibility is to use an HSM algorithm implemented indedicated hardware, for example, an FPGA or ASIC.

Additional embodiments of PCIe-based mass storage systems 10 of thisinvention are represented in FIGS. 4 through 6 as three-tiered storagesystems, in which one or more volatile memory components 32 are includedas an additional storage tier (level) of the mass storage systems 10. Inparticular, FIGS. 4, 5 and 6 represent mass storage systems 10 whosefirst storage tier is provided by an array 44 of volatile memorycomponents 32, while second and third storage tiers of the systems 10are represented by, respectively, two arrays 34 of NAND flash memorycomponents 36 (along with their respective SSD controllers 38) and anarray 40 of miniature hard disk drives 42. It should be noted that theNAND flash memory components 36 and SSD controllers 38 arerepresentative of the memory components and controllers that could beused in the SSD storage devices 20 of FIGS. 1, 2 and 3, and the harddisk drives 42 are representative of the hard disk drives that could beused in the HDD storage devices 20 of FIGS. 1, 2 and 3. In contrast tothe embodiments of FIGS. 1 through 3, all memory components of the threestorage tiers are mounted directly to the surface of the carrier board12. However, it should be appreciated that the removable drive approachrepresented in FIGS. 1 through 3 could be implemented in the embodimentsof FIGS. 4 through 6. In particular, the volatile memory components 32could be in the form factor of a removable single inline memory module(SIMM) or dual inline memory module (DIMM).

In FIG. 4, the NAND flash memory components 36 and their respective SSDcontrollers 38 are addressed through a first RAID controller 14 mountedon the carrier board 12, and the array 40 of hard disk drives 42 areaccessed through a second RAID controller 14 mounted on the carrierboard 12. The embodiment of FIG. 5 is similar to that of FIG. 4, butfurther includes a PCIe switch 46 adapted for interfacing with thesystem logic. As discussed previously in reference to the embodiment ofFIG. 3, the PCIe switch can be used to interface with all PCIe lanes ofthe system 10 at the host system interface level, and then arbitrate thelanes either through physical splitting or through time-divisionmultiplexing. Finally, the embodiment of FIG. 6 is also similar to thatof FIG. 4, but uses a single RAID controller 14 and further includes anHSM manager 48. The mass storage system 10 of FIG. 6 can be used in aneight-channel PCIe configuration with one array in a stripingconfiguration over two SSD storage devices (the two arrays 34 of NANDflash memory components 36) and a second array in a stripingconfiguration over an HDD storage device (the hard disk drives 42) withdistributed parity. Parity calculations for the array 40 of hard diskdrives 42 can be carried out in hardware by the RAID controller 14. Aspreviously noted, the HSM manager 48 may be in the form of an FPGA orASIC.

The volatile memory space of the three-tiered storage systemsrepresented in FIGS. 4-6 can be established through the array 44 ofvolatile memory components 32 being in the form of a suitable module,for example, a standard un-registered dual inline memory module (DIMM),a single inline memory module (SIMM), or a small outline dual inlinememory module SO-DIMM of any suitable design of SDRAM, including thevarious generations of DDR-SDRAM or SGRAM. Addressing and control of thearray 44 of volatile memory components 32 can be implemented usingstandards DRAM control methods and circuits. Alternatively, the volatilememory components 32 can be mounted directly on the carrier board 12 asindividual components, as represented for the volatile memory components16 of FIGS. 1-3.

As another alternative, MRAM or similar nonvolatile memory technologycan be used in place of the volatile memory components 32. MRAMcomponents have similar timing characteristics as DRAM and can beconfigured to have a (DDR) SDRAM-like interface, using time-multiplexingof addresses with simultaneous issuance of standard SDRAM controlsignals. A notable advantage of MRAM is that it is nonvolatile whilebeing comparable with DRAM with respect to write endurance and errorrates.

As noted above, the array 44 of volatile memory components 32functionally serves as the first storage tier of the mass storagesystems 10 of FIGS. 4, 5 and 6. The array 44 provides a large cache sizeof write media, similar to that used on high-end RAID controllers. Morespecifically, the array 44 is functionally comparable to a RAM disk, inthat the array 44 is a volatile extension of the disk space and is partof the virtual memory space of the host system, rather than a partitionof the physical memory of the host system. Especially in any 32-bitoperating system, this has the further advantage of not interfering withany memory address space limitations since the volatile memorycomponents 32 are part of the virtual memory space as opposed to thephysical system memory address space. In most cases, this will have alimited impact since the expected area of usage will be within a 64-bitoperating system.

In the embodiments of FIGS. 4, 5 and 6, all writes from the host systemwill initially go to the array 44 of volatile memory components 32. Thisincludes data and metadata, and takes advantage of the practicallyunlimited write endurance of volatile memory components such as SDRAM.Because of cost and power consumption reasons, it is advantageous to useunregistered modules as the array 44, though in critical applicationsregistered ECC modules may be used. If unregistered SO-DIMMs are used asthe array 44, it is possible to have common power and ground going tothe entire module, but then split the data, address and command busesinto upper 32-bit and lower 32-bit halves and then allocate each half toone of the SSD controllers 38. Alternatively, two SO-DIMMs may be usedto give full independent access of one module to each SSD controller 38.Because the volatile memory components 32 are configured as a cache ofthe nonvolatile NAND flash arrays 34, data can be committed directly tothe NAND flash memory components 36 without accessing the host. Cachemanagement can be done either in software or performed on a custom FPGAor ASIC.

Particularly in the case of a power failure, it is of ultimateimportance to assure that the direct memory dump from the volatile tothe nonvolatile memory domain of the mass storage system 10 can beaccomplished in a relatively short period of time, even if the rest ofthe host system is down, for example, as a consequence of a poweroutage. Back-up power for the system 10 can be supplied by an electricdouble-layer capacitor (EDLC), also known as a super capacitor.Alternatively, the inertia of the spindles of the rotating platters ofthe HDD storage devices 20 (for example, the mini-HDDs of FIGS. 4through 6) may be used to generate enough power to allow for a data dumpfrom the volatile memory components 32 to the NAND flash memorycomponents 36.

In order to warrant a successful dump of the data from the array 44 ofvolatile memory components 32 to the arrays 34 of NAND flash memorycomponents 36, it is also important that fresh flash memory blocks areavailable at least at a capacity equaling the volatile array 44. Theseblocks can be part of an over-provisioning pool of flash memory blocksin the array 36 of NAND flash memory components 36 that are always in anerased state for immediate programming without the need for an erasecycle.

A particular feature available with mass storage systems 10 containingvolatile memory components 32 as the first storage tier of the system 10is the ability to scan compressed files for malicious code, includingviruses and malware. Since compression algorithms are capable of maskingmalicious patterns used for virus detection, it is in most casesnecessary to decompress suspicious files and write them to a tempdirectory in order to scan them. After the scan is completed, the filesin the temp directory are purged. A daily routine system virus scan onan average computer system can result in as much as 20-30 GB/day of datawritten to any storage device for no other purpose than scanning,followed by their immediate deletion. Similar rules apply foressentially any other temporary files, such as installation files, etc.,that are typically purged immediately after the installation of aprogram has been completed. The volatile array 44 as the first tier candramatically reduce overall wear and increase the life span of thenonvolatile array. In this regard, by scanning files when first writtento the first storage tier formed by the volatile memory components 32,the overall life span of the nonvolatile array 36 can be dramaticallyincreased. Accordingly, it is advantageous for temporary files to beautomatically allocated to the volatile memory components 32 of the massstorage system 10.

Similar to the two-tier mass storage systems 10 of FIGS. 1 through 3,the time stamp and access frequency of any file written to one of thethree-tier mass storage systems 10 of FIGS. 4 through 6 can be loggedand, if the data no longer exceed a read access frequency threshold (ormatch some other high priority criterion), they can be purged from thefirst storage tier to either the second or third storage tier of thesystems 10, depending on read access frequency thresholdspre-established for the second and third tiers.

In the case of read accesses, data can be accessed from the finalstorage tier defined by the hard disk drives 42 if they are not storedin one of the first and second storage tiers defined by the volatile andnonvolatile components 32 and 36, respectively. In the case of read-onlydata, for example, document retrieval or applications, the accessfrequency along with the size of the file can be used to determine whichstorage tier the data should be written to. For example, web serverswith high accesses of hot topics can store .html or .php files inmemory. An example would be the startup page of Windows Live™ Messenger,the contents of which are displayed in substantially identical form toall users of Windows Live™ Messenger for a full day without any changes.

If frequently accessed data become obsolete without change in contents,for example, on another day of Messenger contents, a new data set can beloaded from any of the higher storage tiers and there is no need to savethe old data back to their original location. In this case, the data inthe first storage tier are simply updated with the most recent updates.Likewise, in the case of applications that are frequently loaded duringany given time and are therefore copied from the third storage tierdefined by the hard disk drives 42 to one of the first and secondstorage tiers defined by the volatile and nonvolatile components 32 and36, respectively, but where the binaries are not modified, it isunnecessary to write them back to the third storage tier. For example, acomputer game could be frequently loaded until the user has finished thegame. In this case, the application and the current game level would becopied from the third storage tier (hard disk drives 42) to the secondstorage tier (volatile components 36) prior to the first launch of thegame, and as soon as the level has been completed and a new level isloaded, the old level is simply purged from the second storage tier.However, since the original copy is still resident in the third storagetier and the level itself is not modified during game play, there is noneed to write the data back to the hard disk drives 42 of the thirdstorage tier.

In all cases, it is imperative that the file system keeps track of thelocation of the most current file and possible modifications.Accordingly, any move or copy of any data or application file will needto also incur an update of the access path to point to the storagedevice (20, 32, 34 or 40) to which the data or file has been moved orcopied. This can be accomplished through an HSM algorithms (e.g., withthe HSM manager 48 of FIG. 6) using standard methods known in the art.

While the invention has been described in terms of specific embodiments,it is apparent that other forms could be adopted by one skilled in theart. Therefore, the scope of the invention is to be limited only by thefollowing claims.

1. A mass storage system for use in a host system, the mass storagesystem comprising: a printed circuit board having a PCIe edge connectoradapted to interface with a PCIe expansion slot through multiple PCIelanes, the edge connector being adapted to supply power to the massstorage system and enable data exchange between the mass storage systemand a PCIe expansion slot; at least one solid-state mass storage devicehaving nonvolatile solid-state memory components and at least a firstcontroller that interfaces with the nonvolatile solid-state memorycomponents; at least one hard disk mass storage device having at leastone hard disk drive with a rotatable platter; and at least a first RAIDcontroller on the printed circuit board that interfaces with the harddisk mass storage device; wherein the mass storage system is configuredso that the solid-state mass storage device stores files with a readaccess frequency above a threshold, the hard disk mass storage devicestores files with a read access frequency below the threshold, and ifthe read access frequency of a file stored on the hard disk mass storagedevice increases above the threshold, a copy of the file is written tothe solid-state mass storage device.
 2. The mass storage system of claim1, wherein the at least one solid-state mass storage device and the atleast one hard disk mass storage device are removably attached to theprinted circuit board with docking connectors.
 3. The mass storagesystem of claim 1, wherein at least one of the solid-state mass storagedevice and the hard disk mass storage device is integrated into theprinted circuit board.
 4. The mass storage system of claim 1, furthercomprising at least a second RAID controller, wherein the first andsecond RAID controllers interface with different PCIe lanes via the edgeconnector.
 5. The mass storage system of claim 1, further comprising atleast a second RAID controller and a PCIe switch, wherein the first andsecond RAID controllers interface with the edge connector through thePCIe switch.
 6. The mass storage system of claim 5, wherein the firstRAID controller controls a plurality of the hard disk drives in an arrayusing striping with distributed parity and the second RAID controllercontrols at least two arrays of the nonvolatile solid-state memorycomponents in striping mode.
 7. The mass storage system of claim 6,wherein parity calculations for the hard disk drives are carried out inhardware by the first RAID controller.
 8. The mass storage system ofclaim 1, wherein the nonvolatile solid-state memory components arechosen from the group consisting of NAND flash, phase change memory,magnetic RAM, and magneto-resistive RAM.
 9. A mass storage system foruse in a host system, the mass storage system comprising: a printedcircuit board having a PCIe edge connector adapted to interface with aPCIe expansion slot through multiple PCIe lanes, the edge connectorbeing adapted to supply power to the mass storage system and enable dataexchange between the mass storage system and a PCIe expansion slot; atleast one volatile memory module having volatile memory components; atleast one solid-state mass storage device having nonvolatile solid-statememory components and at least a first controller that interfaces withthe nonvolatile solid-state memory components; at least one hard diskmass storage device having at least one hard disk drive with a rotatableplatter; and at least a first RAID controller on the printed circuitboard that interfaces with the hard disk mass storage device; whereinthe mass storage system is configured so that the volatile memory moduleis a cache for the solid-state mass storage device, the solid-state massstorage device permanently stores data with an access frequency above athreshold, the hard disk mass storage device stores data with an accessfrequency below the threshold.
 10. The mass storage system of claim 9,wherein if the read access frequency of data stored on the hard diskmass storage device increases above the threshold, a copy of the data iswritten to the solid-state mass storage device.
 11. The mass storagesystem of claim 9, wherein upon loss of power to the mass storagesystem, the mass storage system is adapted to recover energy from therotatable platter of the hard disk mass storage device to perform amemory dump from the volatile memory module to the solid-state massstorage device.
 12. The mass storage system of claim 9, wherein thesolid-state mass storage device has an over-provisioning pool of atleast the capacity of the volatile memory module.
 13. The mass storagesystem of claim 9, wherein the at least one solid-state mass storagedevice and the at least one hard disk mass storage device are removablyattached to the printed circuit board with docking connectors.
 14. Themass storage system of claim 9, wherein at least one of the solid-statemass storage device and the hard disk mass storage device is integratedinto the printed circuit board.
 15. The mass storage system of claim 9,further comprising at least a second RAID controller, wherein the firstand second RAID controllers interface with different PCIe lanes via theedge connector.
 16. The mass storage system of claim 9, furthercomprising at least a second RAID controller and a PCIe switch, whereinthe first and second RAID controllers interface with the edge connectorthrough the PCIe switch.
 17. The mass storage system of claim 16,wherein the first RAID controller controls a plurality of the hard diskdrives in an array using striping with distributed parity and the secondRAID controller controls at least two arrays of the nonvolatilesolid-state memory components in striping mode.
 18. The mass storagesystem of claim 17, wherein parity calculations for the hard disk drivesare carried out in hardware by the first RAID controller.
 19. The massstorage system of claim 9, wherein the nonvolatile solid-state memorycomponents are chosen from the group consisting of NAND flash, phasechange memory, magnetic RAM, and magneto-resistive RAM.
 20. A method ofusing hierarchical storage management on a mass storage system in a hostsystem, the method comprising: providing a mass storage systemcomprising a printed circuit board, an edge connector adapted tointerface with a PCIe expansion slot through multiple PCIe lanes, atleast one solid-state mass storage device having nonvolatile solid-statememory components and at least a first controller that interfaces withthe nonvolatile solid-state memory components, at least one hard diskmass storage device having at least one hard disk drive with a rotatableplatter, and at least a first RAID controller on the printed circuitboard that interfaces with the hard disk mass storage device;determining a frequency of access for a file stored on the hard diskmass storage device and copying the file to the solid-state mass storagedevice if the access frequency is above a threshold; and updating theaccess path for the file to point to the solid-state mass storagedevice.
 21. The method of claim 20, further comprising removablyinstalling the solid-state mass storage device and the hard disk massstorage device on and removing the solid-state mass storage device andthe hard disk mass storage device from the printed circuit board usingdocking connectors.
 22. The method of claim 20, wherein the mass storagesystem further comprises at least a second RAID controller, and themethod further comprises interfacing the first and second RAIDcontrollers with different PCIe lanes via the edge connector.
 23. Themethod of claim 20, wherein the mass storage system further comprises atleast a second RAID controller, and the method further comprisesinterfacing the first and second RAID controllers with the edgeconnector through a PCIe switch.
 24. The method of claim 23, wherein thefirst RAID controller controls a plurality of the hard disk drives in anarray using striping with distributed parity and the second RAIDcontroller controls at least two arrays of the nonvolatile solid-statememory components in striping mode.
 25. The method of claim 24, furthercomprising carrying out parity calculations for the hard disk driveswith the first RAID controller.
 26. A method of using hierarchicalstorage management on a mass storage system in a host system, the methodcomprising: providing a mass storage system comprising a printed circuitboard, an edge connector adapted to interface with a PCIe expansion slotthrough multiple PCIe lanes, at least one solid-state mass storagedevice having nonvolatile solid-state memory components and at least afirst controller that interfaces with the nonvolatile solid-state memorycomponents, at least one hard disk mass storage device having at leastone hard disk drive with a rotatable platter, at least a first RAIDcontroller on the printed circuit board that interfaces with the harddisk mass storage device, and at least one memory module having memorycomponents associated with the solid-state mass storage device;determining a frequency of access for a file stored on the hard diskmass storage device and copying the file to the solid-state mass storagedevice if the access frequency is above a threshold; and updating theaccess path for the file to point to the solid-state mass storagedevice.
 27. The method of claim 26, wherein the memory module is avolatile memory module, the memory components of the memory module arevolatile memory components, and the solid-state mass storage device hasan over-provisioning pool of at least the capacity of the volatilememory module.
 28. The method of claim 26, further comprising removablyinstalling the solid-state mass storage device and the hard disk massstorage device on and removing the solid-state mass storage device andthe hard disk mass storage device from the printed circuit board usingdocking connectors.
 29. The method of claim 26, wherein upon loss ofpower to the mass storage system, the mass storage system recoversenergy from the rotatable platter of the hard disk mass storage deviceto perform a memory dump from the memory module to the solid-state massstorage device.
 30. The method of claim 26, wherein the mass storagesystem comprises a second solid-state mass storage device havingnonvolatile solid-state memory components and a second controller thatinterfaces with the nonvolatile solid-state memory components thereof,the memory module is a 64-bit wide DRAM memory module configured to haveseparate address and command lines to an upper 32-bit block and a lower32-bit block, the upper 32 bits serve as cache for the first solid-statemass storage device, and the lower 32 bits serve as cache for the secondsolid-state mass storage device.
 31. The method of claim 26, wherein thememory module is a nonvolatile memory module and the memory componentsof the nonvolatile memory module are nonvolatile MRAM components with anSDRAM interface.
 32. A method of using hierarchical storage managementin a host system, the method comprising: providing a mass storage systemcomprising a printed circuit board, an edge connector adapted tointerface with a PCIe expansion slot through multiple PCIe lanes, firstand second RAID controllers interfacing with the edge connector througha PCIe switch, at least two solid-state mass storage devices in astriped array, each of the solid-state mass storage devices havingnonvolatile solid-state memory components functionally connected to thefirst RAID controller, multiple hard disk mass storage devicesfunctionally connected to the second RAID controller, each of the harddisk mass storage devices having at least one hard disk drive with arotatable platter, and a 64-bit wide DRAM memory module as cache for thesolid-state mass storage devices, the memory module configured into afunctional upper 32-bit block and a lower 32-bit block, each of theupper and lower 32-bit blocks having separate command and address buses,the upper 32-bit block being a cache for a first of the solid-state massstorage devices and the lower 32-bit block being a cache for a second ofthe solid-state mass storage devices; copying a file accessed from thehard disk mass storage devices to one of the caches provided by thememory module and updating the access path for the file to point to thesolid-state mass storage devices; storing a copy of the file in thenonvolatile solid-state memory components of the solid-state massstorage devices if the host system writes the file back to the massstorage system; determining a frequency of access of the file; storingthe file to the hard disk mass storage devices if the frequency ofaccess drops below a threshold; and deleting the file from thesolid-state mass storage device and the memory module.